{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "08d9e060-d455-43e2-b1ec-51e2a53e3169",
   "metadata": {},
   "source": [
    "<center><img src=https://raw.githubusercontent.com/feast-dev/feast/master/docs/assets/feast_logo.png width=400/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93095241-3886-44a2-83b1-2a9537c21bc8",
   "metadata": {},
   "source": [
    "# Deploying the Feature Store"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "465783da-18eb-4945-98e7-bb1058a7af1b",
   "metadata": {},
   "source": [
    "### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11961d1b-72db-48dc-a07d-dcea9ba223b4",
   "metadata": {},
   "source": [
    "Feast enables AI/ML teams to serve (and consume) features via feature stores. In this notebook, we will configure the feature stores and feature definitions, and deploy a Feast feature store server. We will also materialize (move) data from the offline store to the online store.\n",
    "\n",
    "In Feast, offline stores support pulling large amounts of data for model training using tools like Redshift, Snowflake, Bigquery, and Spark. In contrast, the focus of Feast online stores is feature serving in support of model inference, using tools like Redis, Snowflake, PostgreSQL, and SQLite.\n",
    "\n",
    "In this notebook, we will setup a file-based (Dask) offline store and SQLite online store. The online store will be made available through the Feast server."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfed8ccf-0d7d-46a1-82f0-5765f8796088",
   "metadata": {},
   "source": [
    "This notebook assumes that you have prepared the data by running the notebook [01_Credit_Risk_Data_Prep.ipynb](01_Credit_Risk_Data_Prep.ipynb). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e66b7a08-5d15-4804-a82a-8bc571777496",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c1e87a4-900b-48f3-a400-ce6608046ce3",
   "metadata": {},
   "source": [
    "*The following code assumes that you have read the example README.md file, and that you have setup an environment where the code can be run. Please make sure you have addressed the prerequisite needs.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8bd21689-4a8e-4b0c-937d-0911df9db1d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import re\n",
    "import sys\n",
    "import time\n",
    "import signal\n",
    "import sqlite3\n",
    "import subprocess\n",
    "import datetime as dt\n",
    "from feast import FeatureStore"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "471db4b0-ea93-47a1-9d55-a80e4d2bdc1e",
   "metadata": {},
   "source": [
    "### Feast Feature Store Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a307490-4121-4bf3-a5c4-77a8885a4f6a",
   "metadata": {},
   "source": [
    "For model training, we usually don't need (or want) a constantly running feature server. All we need is the ability to efficiently query and pull all of the training data at training time. In contrast, during model serving we need servers that are always ready to supply feature records in response to application requests. \n",
    "\n",
    "This training-serving dichotomy is reflected in Feast using \"offline\" and \"online\" stores. Offline stores are configured to work with database technologies typically used for training, while online stores are configured to use storage and streaming technologies that are popular for feature serving.\n",
    "\n",
    "We need to create a `feature_store.yaml` config file to tell feast the structure we want in our offline and online feature stores. Below, we write the configuration for a local \"Dask\" offline store and local SQLite online store. We give the feature store a project name of `loan_applications`, and provider `local`. The registry is where the feature store will keep track of feature definitions and online store updates; we choose a file location in this case.\n",
    "\n",
    "See the [feature_store.yaml](https://docs.feast.dev/reference/feature-repository/feature-store-yaml) documentation for further details. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b3757221-2037-49eb-867f-b9529fec06e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting Feature_Store/feature_store.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile Feature_Store/feature_store.yaml\n",
    "\n",
    "project: loan_applications\n",
    "registry: data/registry.db\n",
    "provider: local\n",
    "offline_store:\n",
    "    type: dask\n",
    "online_store:\n",
    "    type: sqlite\n",
    "    path: data/online_store.db\n",
    "entity_key_serialization_version: 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "180038f3-e5ce-4cce-bdf0-118eee7a822d",
   "metadata": {},
   "source": [
    "### Feature Definitions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd44b206-1f5c-4f55-bbab-41ba2d3f5202",
   "metadata": {},
   "source": [
    "We also need to create feature definitions and other feature constructs in a python file, which we name `feature_definitions.py`. For our purposes, we define the following:\n",
    "\n",
    "- Data Source: connections to data storage or data-producing endpoints\n",
    "- Entity: primary key fields which can be used for joining data\n",
    "- FeatureView: collections of features from a data source\n",
    "\n",
    "For more information on these, see the [Concepts](https://docs.feast.dev/getting-started/concepts) section of the Feast documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d3e8fd80-0bee-463c-b3fb-bd0d1ee83a9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing Feature_Store/feature_definitions.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile Feature_Store/feature_definitions.py\n",
    "\n",
    "# Imports\n",
    "import os\n",
    "from pathlib import Path\n",
    "from feast import (\n",
    "    FileSource,\n",
    "    Entity,\n",
    "    FeatureView,\n",
    "    Field,\n",
    "    FeatureService\n",
    ")\n",
    "from feast.types import Float32, String\n",
    "from feast.data_format import ParquetFormat\n",
    "\n",
    "CURRENT_DIR = os.path.abspath(os.curdir)\n",
    "\n",
    "# Data Sources\n",
    "# A data source tells Feast where the data lives\n",
    "data_a = FileSource(\n",
    "    file_format=ParquetFormat(),\n",
    "    path=Path(CURRENT_DIR,\"data/data_a.parquet\").as_uri()\n",
    ")\n",
    "data_b = FileSource(\n",
    "    file_format=ParquetFormat(),\n",
    "    path=Path(CURRENT_DIR,\"data/data_b.parquet\").as_uri()\n",
    ")\n",
    "\n",
    "# Entity\n",
    "# An entity tells Feast the column it can use to join tables\n",
    "loan_id = Entity(\n",
    "    name = \"loan_id\",\n",
    "    join_keys = [\"ID\"]\n",
    ")\n",
    "\n",
    "# Feature views\n",
    "# A feature view is how Feast groups features\n",
    "features_a = FeatureView(\n",
    "    name=\"data_a\",\n",
    "    entities=[loan_id],\n",
    "    schema=[\n",
    "        Field(name=\"checking_status\", dtype=String),\n",
    "        Field(name=\"duration\", dtype=Float32),\n",
    "        Field(name=\"credit_history\", dtype=String),\n",
    "        Field(name=\"purpose\", dtype=String),\n",
    "        Field(name=\"credit_amount\", dtype=Float32),\n",
    "        Field(name=\"savings_status\", dtype=String),\n",
    "        Field(name=\"employment\", dtype=String),\n",
    "        Field(name=\"installment_commitment\", dtype=Float32),\n",
    "        Field(name=\"personal_status\", dtype=String),\n",
    "        Field(name=\"other_parties\", dtype=String),\n",
    "    ],\n",
    "    source=data_a\n",
    ")\n",
    "features_b = FeatureView(\n",
    "    name=\"data_b\",\n",
    "    entities=[loan_id],\n",
    "    schema=[\n",
    "        Field(name=\"residence_since\", dtype=Float32),\n",
    "        Field(name=\"property_magnitude\", dtype=String),\n",
    "        Field(name=\"age\", dtype=Float32),\n",
    "        Field(name=\"other_payment_plans\", dtype=String),\n",
    "        Field(name=\"housing\", dtype=String),\n",
    "        Field(name=\"existing_credits\", dtype=Float32),\n",
    "        Field(name=\"job\", dtype=String),\n",
    "        Field(name=\"num_dependents\", dtype=Float32),\n",
    "        Field(name=\"own_telephone\", dtype=String),\n",
    "        Field(name=\"foreign_worker\", dtype=String),\n",
    "    ],\n",
    "    source=data_b\n",
    ")\n",
    "\n",
    "# Feature Service\n",
    "# a feature service in Feast represents a logical group of features\n",
    "loan_fs = FeatureService(\n",
    "    name=\"loan_fs\",\n",
    "    features=[features_a, features_b]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b47c1b5-849e-43f3-8043-60466aaed69f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "be9723eb-8fa0-4338-b50c-f9f1ff6bb13a",
   "metadata": {},
   "source": [
    "### Applying the Configuration and Definitions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c796d45f-28c0-4875-bbb1-71e5a15dcb96",
   "metadata": {},
   "source": [
    "Now that we have our feature store configuration (`feature_store.yaml`) and feature definitions (`feature_definitions.py`), we are ready to \"apply\" them. The `feast apply` command creates a registry file (`Feature_Store/data/registry.db`) and sets up data connections; in this case, it creates a SQLite database (`Feature_Store/data/online_store.db`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "394467f3-4ced-492a-9379-105aea9d4a6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10/27/2024 02:19:03 PM root WARNING: Cannot use sqlite_vec for vector search\n",
      "10/27/2024 02:19:03 PM root WARNING: Cannot use sqlite_vec for vector search\n",
      "10/27/2024 02:19:03 PM root WARNING: Cannot use sqlite_vec for vector search\n",
      "10/27/2024 02:19:03 PM root WARNING: Cannot use sqlite_vec for vector search\n",
      "Created entity \u001b[1m\u001b[32mloan_id\u001b[0m\n",
      "Created feature view \u001b[1m\u001b[32mdata_a\u001b[0m\n",
      "Created feature view \u001b[1m\u001b[32mdata_b\u001b[0m\n",
      "Created feature service \u001b[1m\u001b[32mloan_fs\u001b[0m\n",
      "\n",
      "10/27/2024 02:19:03 PM root WARNING: Cannot use sqlite_vec for vector search\n",
      "10/27/2024 02:19:03 PM root WARNING: Cannot use sqlite_vec for vector search\n",
      "Created sqlite table \u001b[1m\u001b[32mloan_applications_data_a\u001b[0m\n",
      "Created sqlite table \u001b[1m\u001b[32mloan_applications_data_b\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Run 'feast apply' in the Feature_Store directory\n",
    "!feast --chdir ./Feature_Store apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e32f40eb-a31a-4877-8f40-2d8515302f39",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 232\n",
      "-rw-r--r--  1 501  20    33K Oct 27 14:17 data_a.parquet\n",
      "-rw-r--r--  1 501  20    27K Oct 27 14:17 data_b.parquet\n",
      "-rw-r--r--  1 501  20    17K Oct 27 14:17 labels.parquet\n",
      "-rw-r--r--  1 501  20    28K Oct 27 14:19 online_store.db\n",
      "-rw-r--r--  1 501  20   2.8K Oct 27 14:19 registry.db\n"
     ]
    }
   ],
   "source": [
    "# List the Feature_Store/data/ directory to see newly created files\n",
    "!ls -nlh Feature_Store/data/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31014885-ce6a-4007-8bdb-d74d3b44781b",
   "metadata": {},
   "source": [
    "Note that while `feast apply` set up the `sqlite` online database, `online_store.db`, no data has been added to the online database as of yet. We can verify this by connecting with the `sqlite3` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "107ca856-af06-40c4-8339-70daf59cdf37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Online Store Tables:            [('loan_applications_data_a',), ('loan_applications_data_b',)]\n",
      "loan_applications_data_a data:  []\n",
      "loan_applications_data_b data:  []\n"
     ]
    }
   ],
   "source": [
    "# Connect to sqlite database\n",
    "conn = sqlite3.connect(\"Feature_Store/data/online_store.db\")\n",
    "cursor = conn.cursor()\n",
    "# Query table data (3 tables)\n",
    "print(\n",
    "    \"Online Store Tables:           \",\n",
    "    cursor.execute(\"SELECT name FROM sqlite_master WHERE type='table';\").fetchall()\n",
    ")\n",
    "print(\n",
    "    \"loan_applications_data_a data: \",\n",
    "    cursor.execute(\"SELECT * FROM loan_applications_data_a\").fetchall()\n",
    ")\n",
    "print(\n",
    "    \"loan_applications_data_b data: \",\n",
    "    cursor.execute(\"SELECT * FROM loan_applications_data_b\").fetchall()\n",
    ")\n",
    "conn.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03b927ee-7913-4a8a-b17b-9bee361d8d94",
   "metadata": {},
   "source": [
    "Since we have used `feast apply` to create the registry, we can now use the Feast Python SDK to interact with our new feature store. To see other possible commands see the [Feast Python SDK documentation](https://rtd.feast.dev/en/master/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c764a60a-b911-41a8-ba8f-7ef0a0bc7257",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jyejare/gitrepos/feast/.venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:192: UserWarning: Field name \"vector_enabled\" in \"SqliteOnlineStoreConfig\" shadows an attribute in parent \"VectorStoreConfig\"\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RepoConfig(project='loan_applications', provider='local', registry_config='data/registry.db', online_config={'type': 'sqlite', 'path': 'data/online_store.db'}, auth={'type': 'no_auth'}, offline_config={'type': 'dask'}, batch_engine_config='local', feature_server=None, flags=None, repo_path=PosixPath('Feature_Store'), entity_key_serialization_version=3, coerce_tz_aware=True)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get feature store config\n",
    "store = FeatureStore(repo_path=\"./Feature_Store\")\n",
    "store.config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fc572976-6ce9-44f6-8b67-28ee6157e29c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature view: data_a  |  Features: [checking_status-String, duration-Float32, credit_history-String, purpose-String, credit_amount-Float32, savings_status-String, employment-String, installment_commitment-Float32, personal_status-String, other_parties-String]\n",
      "Feature view: data_b  |  Features: [residence_since-Float32, property_magnitude-String, age-Float32, other_payment_plans-String, housing-String, existing_credits-Float32, job-String, num_dependents-Float32, own_telephone-String, foreign_worker-String]\n"
     ]
    }
   ],
   "source": [
    "# List feature views\n",
    "feature_views = store.list_batch_feature_views()\n",
    "for fv in feature_views:\n",
    "    print(f\"Feature view: {fv.name}  |  Features: {fv.features}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "027edcfe-58d7-4dcb-92e2-5a5514c0f1f0",
   "metadata": {},
   "source": [
    "### Deploying the Feature Store Servers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9aab68d-395f-421e-ba11-ad8c4acc9d6f",
   "metadata": {},
   "source": [
    "If you wish to share a feature store with your team, Feast provides feature servers. To spin up an offline feature server process, we can use the `feast serve_offline` command, while to spin up a Feast online feature server, we use the `feast serve` command.\n",
    "\n",
    "Let's spin up an offline and an online server that we can use in the subsequent notebooks to get features during model training and model serving. We will run both servers as background processes, that we can communicate with in the other notebooks.\n",
    "\n",
    "First, we write a helper function to extract the first few printed log lines (so we can print it in the notebook cell output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "568f81b8-df34-4b06-8a3f-1a6bdc2e6cff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TimeoutError class\n",
    "class TimeoutError(Exception):\n",
    "    pass\n",
    "\n",
    "# TimeoutError raise function\n",
    "def timeout():\n",
    "    raise TimeoutError(\"timeout\")\n",
    "\n",
    "# Get first few log lines function\n",
    "def print_first_proc_lines(proc, wait):\n",
    "    '''Given a process, `proc`, read and print output lines until they stop \n",
    "    comming (waiting up to `wait` seconds for new lines to appear)'''\n",
    "    lines = \"\"\n",
    "    while True:\n",
    "        signal.signal(signal.SIGALRM, timeout)\n",
    "        signal.alarm(wait)\n",
    "        try:\n",
    "            lines += proc.stderr.readline()\n",
    "        except:\n",
    "            break\n",
    "    if lines:\n",
    "        print(lines, file=sys.stderr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88d25a87-241a-46c6-9ca7-d035959c5f74",
   "metadata": {},
   "source": [
    "Launch the offline server with the command `feast --chdir ./Feature_Store serve_offline`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ce965dd4-652b-4c36-a064-fd0fd97d3ef7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feast offline server process\n",
    "offline_server_proc = subprocess.Popen(\n",
    "    \"feast --chdir ./Feature_Store serve_offline 2>&2 & echo $! > server_proc.txt\",\n",
    "    shell=True,\n",
    "    text=True,\n",
    "    stdout=subprocess.PIPE,\n",
    "    stderr=subprocess.PIPE,\n",
    "    bufsize=0\n",
    ")\n",
    "print_first_proc_lines(offline_server_proc, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59958d64-8e68-45ff-9549-556cbf46908c",
   "metadata": {},
   "source": [
    "The tail end of the command above, `2>&2 & echo $! > server_proc.txt`, captures log messages (in the offline case there are none), and writes the process PID to the file `server_proc.txt` (we will use this in the cleanup notebook, [05_Credit_Risk_Cleanup.ipynb](05_Credit_Risk_Cleanup.ipynb))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfed4334-9e62-4f3f-be96-3f7db2f06ada",
   "metadata": {},
   "source": [
    "Next, launch the online server with the command `feast --chdir ./Feature_Store serve`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a581fbe2-13ba-433e-8e76-dc82cc22af74",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ddowler/Code/Feast/feast/examples/credit-risk-end-to-end/venv-py3.11/lib/python3.11/site-packages/uvicorn/workers.py:16: DeprecationWarning: The `uvicorn.workers` module is deprecated. Please use `uvicorn-worker` package instead.\n",
      "For more details, see https://github.com/Kludex/uvicorn-worker.\n",
      "  warnings.warn(\n",
      "[2024-10-27 14:19:07 -0600] [44621] [INFO] Starting gunicorn 23.0.0\n",
      "[2024-10-27 14:19:07 -0600] [44621] [INFO] Listening at: http://127.0.0.1:6566 (44621)\n",
      "[2024-10-27 14:19:07 -0600] [44621] [INFO] Using worker: uvicorn.workers.UvicornWorker\n",
      "[2024-10-27 14:19:07 -0600] [44623] [INFO] Booting worker with pid: 44623\n",
      "[2024-10-27 14:19:07 -0600] [44623] [INFO] Started server process [44623]\n",
      "[2024-10-27 14:19:07 -0600] [44623] [INFO] Waiting for application startup.\n",
      "[2024-10-27 14:19:07 -0600] [44623] [INFO] Application startup complete.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Feast online server (master and worker) processes\n",
    "online_server_proc = subprocess.Popen(\n",
    "    \"feast --chdir ./Feature_Store serve 2>&2 & echo $! >> server_proc.txt\",\n",
    "    shell=True,\n",
    "    text=True,\n",
    "    stdout=subprocess.PIPE,\n",
    "    stderr=subprocess.PIPE,\n",
    "    bufsize=0\n",
    ")\n",
    "print_first_proc_lines(online_server_proc, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e778173-f58a-4074-b63f-107e1f39577b",
   "metadata": {},
   "source": [
    "Note that the output helpfully let's us know that the online server is \"Listening at: http://127.0.0.1:6566\" (the default host:port).\n",
    "\n",
    "List the running processes to verify they are up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9b1a224d-884d-45c5-9711-2e2eb4351710",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  501 44594     1   0  2:19PM ??         0:03.66 **/python **/feast --chdir ./Feature_Store serve_offline\n",
      "  501 44621     1   0  2:19PM ??         0:03.58 **/python **/feast --chdir ./Feature_Store serve\n",
      "  501 44623 44621   0  2:19PM ??         0:00.03 **/python **/feast --chdir ./Feature_Store serve\n",
      "  501 44662 44542   0  2:19PM ??         0:00.01 /bin/zsh -c ps -ef | grep **/feast | grep serve\n"
     ]
    }
   ],
   "source": [
    "# List running Feast processes (paths redacted)\n",
    "running_procs = !ps -ef | grep feast | grep serve\n",
    "\n",
    "for line in running_procs:\n",
    "    redacted = re.sub(r'/*[^\\s]*(?P<cmd>(python )|(feast ))', r'**/\\g<cmd>', line)\n",
    "    print(redacted)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd52eeb4-948c-472b-9111-8549fda955a1",
   "metadata": {},
   "source": [
    "Note that there are two process for the online server (master and worker)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8258e7a8-5f6e-4737-93ee-63591518b169",
   "metadata": {},
   "source": [
    "### Materialize Features to the Online Store"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21b354ab-ec22-476d-8fd9-6ffe0f3fbacb",
   "metadata": {},
   "source": [
    "At this point, there is no data in the online store yet. Let's use the SDK feature store object (that we created above) to \"materialize\" data; this is Feast lingo for moving/updating data from the offline store to the online store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ff6146df-03a7-4ac2-a665-ee5f440c3605",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:_list_feature_views will make breaking changes. Please use _list_batch_feature_views instead. _list_feature_views will behave like _list_all_feature_views in the future.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Materializing \u001b[1m\u001b[32m2\u001b[0m feature views from \u001b[1m\u001b[32m2023-09-24 12:00:00-06:00\u001b[0m to \u001b[1m\u001b[32m2024-01-07 12:00:00-07:00\u001b[0m into the \u001b[1m\u001b[32msqlite\u001b[0m online store.\n",
      "\n",
      "\u001b[1m\u001b[32mdata_a\u001b[0m:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|                                                                      | 0/1000 [00:00<?, ?it/s]WARNING:root:Cannot use sqlite_vec for vector search\n",
      "100%|████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 14867.67it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[32mdata_b\u001b[0m:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 9395.32it/s]\n"
     ]
    }
   ],
   "source": [
    "# Materialize\n",
    "# Recall that we mocked the outcome data to have timestamps from \n",
    "# 'Tue Sep 24 12:00:00 2023'out to \"Wed Oct  9 12:00:00 2023\"\n",
    "# The loan outcome timestamps were then lagged by 30-90 days (which is Jan 7 12:00:00 2024)\n",
    "res = store.materialize(\n",
    "    start_date=dt.datetime(2023,9,24,12,0,0),\n",
    "    end_date=dt.datetime(2024,1,7,12,0,0)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c053fd7b-336d-4286-a3aa-80efee43d2d6",
   "metadata": {},
   "source": [
    "Now, we can query the SQLite database again and see data in the response!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "920f8427-5211-4c0b-9873-0bf42f14aefb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loan_applications_data_a data:  [(b'\\x02\\x00\\x00\\x00ID\\x04\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x0b\\x01\\x00\\x00\\x00\\x00\\x00\\x00', 'checking_status', b'\\x12\\x0bno checking', None, '2023-09-24 12:04:20', None), (b'\\x02\\x00\\x00\\x00ID\\x04\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x0b\\x01\\x00\\x00\\x00\\x00\\x00\\x00', 'duration', b'5\\x00\\x00\\xc0A', None, '2023-09-24 12:04:20', None)]\n",
      "loan_applications_data_b data:  [(b'\\x02\\x00\\x00\\x00ID\\x04\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x0b\\x01\\x00\\x00\\x00\\x00\\x00\\x00', 'residence_since', b'5\\x00\\x00@@', None, '2023-09-24 12:04:20', None), (b'\\x02\\x00\\x00\\x00ID\\x04\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x0b\\x01\\x00\\x00\\x00\\x00\\x00\\x00', 'property_magnitude', b'\\x12\\x03car', None, '2023-09-24 12:04:20', None)]\n"
     ]
    }
   ],
   "source": [
    "# Query the online store database to verify materialized data\n",
    "conn = sqlite3.connect(\"Feature_Store/data/online_store.db\")\n",
    "cursor = conn.cursor()\n",
    "print(\n",
    "    \"loan_applications_data_a data: \",\n",
    "    cursor.execute(\"SELECT * FROM loan_applications_data_a LIMIT 2\").fetchall()\n",
    ")\n",
    "print(\n",
    "    \"loan_applications_data_b data: \",\n",
    "    cursor.execute(\"SELECT * FROM loan_applications_data_b LIMIT 2\").fetchall()\n",
    ")\n",
    "conn.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24c6221e-3914-4f2e-a922-d2263c6df5f7",
   "metadata": {},
   "source": [
    "Note that the data is stored in binary strings, which is part of Feast's optimization for online queries. To get human-readable data, use the `get-online-features` REST API command, which returns a JSON response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "fcd1beb0-fc13-4c4b-9396-ca58eb7f02af",
   "metadata": {},
   "outputs": [],
   "source": [
    "# curl command to online server to get data from the online store\n",
    "cmd = \"\"\"http://localhost:6566/get-online-features \\\n",
    "    -d '{ \n",
    "            \"feature_service\": \"loan_fs\",\n",
    "            \"entities\": {\"ID\": [18, 764]}\n",
    "        }'\n",
    "\"\"\"\n",
    "\n",
    "response = !curl -X POST {cmd}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c8b3ea8d-1270-44f2-9a5c-1d8b54b36b2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current',\n",
       " '                                 Dload  Upload   Total   Spent    Left  Speed',\n",
       " '',\n",
       " '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0',\n",
       " '100  3215  100  3119  100    96   266k   8396 --:--:-- --:--:-- --:--:--  261k',\n",
       " '100  3215  100  3119  100    96   265k   8371 --:--:-- --:--:-- --:--:--  261k',\n",
       " '{\"metadata\":{\"feature_names\":[\"ID\",\"savings_status\",\"employment\",\"checking_status\",\"credit_history\",\"personal_status\",\"credit_amount\",\"other_parties\",\"purpose\",\"duration\",\"installment_commitment\",\"own_telephone\",\"residence_since\",\"num_dependents\",\"age\",\"other_payment_plans\",\"housing\",\"existing_credits\",\"job\",\"property_magnitude\",\"foreign_worker\"]},\"results\":[{\"values\":[18,764],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"1970-01-01T00:00:00Z\",\"1970-01-01T00:00:00Z\"]},{\"values\":[\"<100\",\"100<=X<500\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\">=7\",\"4<=X<7\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"0<=X<200\",\"no checking\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"existing paid\",\"critical/other existing credit\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"female div/dep/mar\",\"male mar/wid\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[12579.0,2463.0],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"none\",\"none\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"used car\",\"new car\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[24.0,24.0],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[4.0,4.0],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"yes\",\"yes\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[2.0,3.0],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[1.0,1.0],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[44.0,27.0],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"none\",\"none\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"for free\",\"own\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[1.0,2.0],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"high qualif/self emp/mgmt\",\"skilled\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"no known property\",\"life insurance\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]},{\"values\":[\"yes\",\"yes\"],\"statuses\":[\"PRESENT\",\"PRESENT\"],\"event_timestamps\":[\"2023-09-25T01:03:47Z\",\"2023-09-29T03:17:24Z\"]}]}']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01d20196-1d42-486d-a0bd-97193c953785",
   "metadata": {},
   "source": [
    "The `curl` command gave us a quick validation. In the [04_Credit_Risk_Model_Serving.ipynb](04_Credit_Risk_Model_Serving.ipynb) notebook, we'll use the Python `requests` library to handle the query better."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74a5117-dd34-4dde-93a8-ea6e8c4c545a",
   "metadata": {},
   "source": [
    "Now that the feature stores and their respective servers have been configured and deployed, we can proceed to train an AI model in [03_Credit_Risk_Model_Training.ipynb](03_Credit_Risk_Model_Training.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
